Deception detection system and method

ABSTRACT

A system for detecting deception is provided. The system comprises a camera, an image processing unit, and a notification device. The camera is configured to capture an image sequence of a person of interest. The image processing unit is trained to determine a set of bitplanes of a plurality of images in the captured image sequence that represent the hemoglobin concentration (HC) changes of the person, and to detect the person&#39;s invisible emotional states based on HC changes. The image processing unit is trained using a training set comprising a set of subjects for which emotional state is known. The notification device provides a notification of at least one of the person&#39;s detected invisible emotional states.

TECHNICAL FIELD

The following relates generally to security and more specifically to animage-capture based system and method for detecting deception.

BACKGROUND

Security has been considered a relatively-high concern for a number ofrecent years. Of particular interest is security afforded tocheckpoints, such as border crossings, airport security checkpoints,sensitive building entrances, etc. While it is desirable to provide verythorough security, there is a balance between the security afforded andthe comprehensive costs associated therewith. There are a few main costsfor providing security at such checkpoints: manpower, efficiency, and,as a result, use.

In order to understand the threat that a person poses, the person can besearched in an attempt to identify the threat, or the person can beinterviewed. This latter option is generally more rapid and lessinvasive, leading to a lower level of dissatisfaction inusers/customers. Security staff at security checkpoints is trained toidentify possible deception based on visual cues and audio cues. Aperson, however, can condition themselves to reduce thehuman-perceptible signs of deception, such as twitches, fidgeting,wavering in the voice, a break in eye contact, etc. As a result, whilestaff training can lead to the detection of some deception, it isunlikely to lead to the detection of deception for more egregious cases.

The detection of deception and hidden emotions generally is of interestin other circumstances, such as during the interrogation of a suspect ora potential witness of a crime, or the surveillance of a person ofinterest. In such cases, it can be helpful to identify hidden orotherwise obscured emotions that can provide insight into the veracityof answers provided, the anxiousness/discomfort of a person, etc.

SUMMARY

In one aspect, a system for detecting deception for the securityscreening of a person of interest by an attendant, is provided, thesystem comprising: a camera configured to capture an image sequence ofthe person of interest; a processing unit trained to determine a set ofbitplanes of a plurality of images in the captured image sequence thatrepresent the hemoglobin concentration (HC) changes of the person, todetect the person's invisible emotional states based on HC changes, andto output the detected invisible emotional states, the processing unitbeing trained using a training set comprising HC changes of subjectswith known emotional states; and, a notification device for providing anotification of at least one of the person's detected invisibleemotional states to the attendant based on the output of the processingunit.

In another aspect, a method for detecting deception for the securityscreening of a person of interest by an attendant, is provided, themethod comprising: capturing, by a camera, an image sequence of theperson of interest; determining, by a processing unit, a set ofbitplanes of a plurality of images in the captured image sequence thatrepresent the hemoglobin concentration (HC) changes of the person,detecting the person's invisible emotional states based on HC changes,and outputting the detected invisible emotional states, the processingunit being trained using a training set comprising HC changes ofsubjects with known emotional states; and, providing a notification, bya notification device, of at least one of the person's detectedinvisible emotional states to the attendant based on the output of theprocessing unit.

BRIEF DESCRIPTION OF THE DRAWINGS

The features of the invention will become more apparent in the followingdetailed description in which reference is made to the appended drawingswherein:

FIG. 1 is a top-down section view of a system for detecting deceptionemployed at a border security checkpoint in accordance with anembodiment;

FIG. 2 is an block diagram of various components of the system fordeception detection of FIG. 1;

FIG. 3 illustrates re-emission of light from skin epidermal andsubdermal layers;

FIG. 4 is a set of surface and corresponding transdermal imagesillustrating change in hemoglobin concentration associated withinvisible emotion for a particular human subject at a particular pointin time;

FIG. 5 is a plot illustrating hemoglobin concentration changes for theforehead of a subject who experiences positive, negative, and neutralemotional states as a function of time (seconds);

FIG. 6 is a plot illustrating hemoglobin concentration changes for thenose of a subject who experiences positive, negative, and neutralemotional states as a function of time (seconds);

FIG. 7 is a plot illustrating hemoglobin concentration changes for thecheek of a subject who experiences positive, negative, and neutralemotional states as a function of time (seconds);

FIG. 8 is a flowchart illustrating a fully automated transdermal opticalimaging and invisible emotion detection system;

FIG. 9 is an exemplary screen presented to the border security guard bythe computer system via the display of FIG. 1;

FIG. 10 is an illustration of a data-driven machine learning system foroptimized hemoglobin image composition;

FIG. 11 is an illustration of a data-driven machine learning system formultidimensional invisible emotion model building;

FIG. 12 is an illustration of an automated invisible emotion detectionsystem;

FIG. 13 is a memory cell; and

FIG. 14 is a kiosk for presenting a questionnaire, and having a camerafor detecting deception in accordance with another embodiment.

DETAILED DESCRIPTION

Embodiments will now be described with reference to the figures. Forsimplicity and clarity of illustration, where considered appropriate,reference numerals may be repeated among the Figures to indicatecorresponding or analogous elements. In addition, numerous specificdetails are set forth in order to provide a thorough understanding ofthe embodiments described herein. However, it will be understood bythose of ordinary skill in the art that the embodiments described hereinmay be practiced without these specific details. In other instances,well-known methods, procedures and components have not been described indetail so as not to obscure the embodiments described herein. Also, thedescription is not to be considered as limiting the scope of theembodiments described herein.

Various terms used throughout the present description may be read andunderstood as follows, unless the context indicates otherwise: “or” asused throughout is inclusive, as though written “and/or”; singulararticles and pronouns as used throughout include their plural forms, andvice versa; similarly, gendered pronouns include their counterpartpronouns so that pronouns should not be understood as limiting anythingdescribed herein to use, implementation, performance, etc. by a singlegender; “exemplary” should be understood as “illustrative” or“exemplifying” and not necessarily as “preferred” over otherembodiments. Further definitions for terms may be set out herein; thesemay apply to prior and subsequent instances of those terms, as will beunderstood from a reading of the present description.

Any module, unit, component, server, computer, terminal, engine ordevice exemplified herein that executes instructions may include orotherwise have access to computer readable media such as storage media,computer storage media, or data storage devices (removable and/ornon-removable) such as, for example, magnetic disks, optical disks, ortape. Computer storage media may include volatile and non-volatile,removable and non-removable media implemented in any method ortechnology for storage of information, such as computer readableinstructions, data structures, program modules, or other data. Examplesof computer storage media include RAM, ROM, EEPROM, flash memory orother memory technology, CD-ROM, digital versatile disks (DVD) or otheroptical storage, magnetic cassettes, magnetic tape, magnetic diskstorage or other magnetic storage devices, or any other medium which canbe used to store the desired information and which can be accessed by anapplication, module, or both. Any such computer storage media may bepart of the device or accessible or connectable thereto. Further, unlessthe context clearly indicates otherwise, any processor or controller setout herein may be implemented as a singular processor or as a pluralityof processors. The plurality of processors may be arrayed ordistributed, and any processing function referred to herein may becarried out by one or by a plurality of processors, even though a singleprocessor may be exemplified. Any method, application or module hereindescribed may be implemented using computer readable/executableinstructions that may be stored or otherwise held by such computerreadable media and executed by the one or more processors.

The following relates generally to a system for detecting deception anda method therefor. A specific embodiment relates to an image-capturebased system and method for detecting deception at a securitycheckpoint, and specifically the invisible emotional state of anindividual captured in a series of images or a video. The systemprovides a remote and non-invasive approach by which to detect deceptionwith a high confidence.

It can be desirable to determine the emotional state of a person todetect deception and/or discomfort. For example, a person passing asecurity checkpoint may be unusually uncomfortable with the experience,or may hide or alter the truth when answering a question of securitycheckpoint staff or a question posed in a checkpoint machine. It can berelatively easy to control one's visible emotional state, but verydifficult to mask physiological changes corresponding to emotional statechanges. The detected invisible emotion can be used by securitycheckpoint staff to make decisions regarding the further investigationof a person passing through.

FIG. 1 shows a system 20 for detecting deception at a border securitycheckpoint in accordance with an embodiment. A vehicle 24 is shownhaving a driver 28 positioned inside in a driver's seat. The vehicle 24is pulled up to a border security checkpoint station 32. The system 20is deployed inside the border security checkpoint station 32, andcomprises a computer system 36, a display 40 which is shown as angled tobe visible only to a border security guard 44 and not to the driver 28,and a camera 48 coupled to the computer system 36 via a wired orwireless communication medium, such as Ethernet, Universal Serial Bus(“USB”), IEEE 802.11 (“Wi-Fi”), Bluetooth, etc.

The camera 48 is configured to capture image sequences of particularbody parts of the driver 28. Typically, the driver's 28 face will becaptured. The camera 38 can be any suitable visible light camera typefor capturing an image sequence of a consumer's face, such as, forexample, a CMOS or CCD camera. The camera 48 can be configured withlenses to enable image capture from a wider angle, and the computersystem 34 can be configured to transform the image sequences tocompensate for any distorsion introduced by the lenses.

Hemoglobin concentration (HC) can be isolated from raw images taken fromthe camera 38, and spatial-temporal changes in HC can be correlated tohuman emotion. Referring now to FIG. 3, a diagram illustrating there-emission of light from skin is shown. Light (201) travels beneath theskin (202), and re-emits (203) after travelling through different skintissues. The re-emitted light (203) may then be captured by opticalcameras. The dominant chromophores affecting the re-emitted light aremelanin and hemoglobin. Since melanin and hemoglobin have differentcolor signatures, it has been found that it is possible to obtain imagesmainly reflecting HC under the epidermis as shown in FIG. 4.

The system 20 implements a two-step method to generate rules suitable tooutput an estimated statistical probability that a human subject'semotional state belongs to one of a plurality of emotions, and anormalized intensity measure of such emotional state given a videosequence of any subject. The emotions detectable by the systemcorrespond to those for which the system is trained.

Referring now to FIG. 2, various components of the system 20 fordeception detection at a security checkpoint are shown in isolation. Thecomputer system 34 comprises an image processing unit 104, an imagefilter 106, an image classification machine 105, and a storage device101. A processor of the computer system 34 retrieves computer-readableinstructions from the storage device 101 and executes them to implementthe image processing unit 104, the image filter 106, and the imageclassification machine 105, The image classification machine 105 isconfigured with training configuration data 102 derived from anothercomputer system trained using a training set of images and is operableto perform classification for a query set of images 103 which aregenerated from images captured by the camera 38, processed by the imagefilter 106, and stored on the storage device 102.

The sympathetic and parasympathetic nervous systems are responsive toemotion. It has been found that an individual's blood flow is controlledby the sympathetic and parasympathetic nervous system, which is beyondthe conscious control of the vast majority of individuals. Thus, anindividual's internally experienced emotion can be readily detected bymonitoring their blood flow. Internal emotion systems prepare humans tocope with different situations in the environment by adjusting theactivations of the autonomic nervous system (ANS); the sympathetic andparasympathetic nervous systems play different roles in emotionregulation with the former regulating up fight-flight reactions whereasthe latter serves to regulate down the stress reactions. Basic emotionshave distinct ANS signatures. Blood flow in most parts of the face suchas eyelids, cheeks and chin is predominantly controlled by thesympathetic vasodilator neurons, whereas blood flowing in the nose andears is mainly controlled by the sympathetic vasoconstrictor neurons; incontrast, the blood flow in the forehead region is innervated by bothsympathetic and parasympathetic vasodilators. Thus, different internalemotional states have differential spatial and temporal activationpatterns on the different parts of the face. By obtaining hemoglobindata from the system, facial hemoglobin concentration (HC) changes invarious specific facial areas may be extracted. These multidimensionaland dynamic arrays of data from an individual are then compared tocomputational models based on normative data to be discussed in moredetail below. From such comparisons, reliable statistically basedinferences about an individual's internal emotional states may be made.Because facial hemoglobin activities controlled by the ANS are notreadily subject to conscious controls, such activities provide anexcellent window into an individual's genuine innermost emotions.

Referring now to FIG. 8, a flowchart illustrating the method ofinvisible emotion detection performed by the system 20 is shown. Thesystem 20 performs image registration 701 to register the input of avideo/image sequence captured of a subject with an unknown emotionalstate, hemoglobin image extraction 702, ROI selection 703, multi-ROIspatial-temporal hemoglobin data extraction 704, invisible emotion model705 application, data mapping 706 for mapping the hemoglobin patterns ofchange, emotion detection 707, and notification 708. FIG. 12 depictsanother such illustration of automated invisible emotion detectionsystem.

The image processing unit obtains each captured image or video streamfrom each camera and performs operations upon the image to generate acorresponding optimized HC image of the subject. The image processingunit isolates HC in the captured video sequence. In an exemplaryembodiment, the images of the subject's faces are taken at 30 frames persecond using the camera. It will be appreciated that this process may beperformed with alternative digital cameras and lighting conditions.

Isolating HC is accomplished by analyzing bitplanes in the videosequence to determine and isolate a set of the bitplanes that providehigh signal to noise ratio (SNR) and, therefore, optimize signaldifferentiation between different emotional states on the facialepidermis (or any part of the human epidermis). The determination ofhigh SNR bitplanes is made with reference to a first training set ofimages constituting the captured video sequence, coupled with EKG,pneumatic respiration, blood pressure, laser Doppler data from the humansubjects from which the training set is obtained. The EKG and pneumaticrespiration data are used to remove cardiac, respiratory, and bloodpressure data in the HC data to prevent such activities from masking themore-subtle emotion-related signals in the HC data. The second stepcomprises training a machine to build a computational model for aparticular emotion using spatial-temporal signal patterns of epidermalHC changes in regions of interest (“ROIs”) extracted from the optimized“bitplaned” images of a large sample of human subjects.

For training, video images of test subjects exposed to stimuli known toelicit specific emotional responses are captured. Responses may begrouped broadly (neutral, positive, negative) or more specifically(distressed, happy, anxious, sad, frustrated, intrigued, joy, disgust,angry, surprised, contempt, etc.). In further embodiments, levels withineach emotional state may be captured. Preferably, subjects areinstructed not to express any emotions on the face so that the emotionalreactions measured are invisible emotions and isolated to changes in HC.To ensure subjects do not “leak” emotions in facial expressions, thesurface image sequences may be analyzed with a facial emotionalexpression detection program. EKG, pneumatic respiratory, bloodpressure, and laser Doppler data may further be collected using an EKGmachine, a pneumatic respiration machine, a continuous blood pressuremachine, and a laser Doppler machine and provides additional informationto reduce noise from the bitplane analysis, as follows.

ROIs for emotional detection (e.g., forehead, nose, and cheeks) aredefined manually or automatically for the video images. These ROIs arepreferably selected on the basis of knowledge in the art in respect ofROIs for which HC is particularly indicative of emotional state. Usingthe native images that consist of all bitplanes of all three R, G, Bchannels, signals that change over a particular time period (e.g., 10seconds) on each of the ROIs in a particular emotional state (e.g.,positive) are extracted. The process may be repeated with otheremotional states (e.g., negative or neutral). The EKG and pneumaticrespiration data may be used to filter out the cardiac, respirator, andblood pressure signals on the image sequences to prevent non-emotionalsystemic HC signals from masking true emotion-related HC signals. FastFourier transformation (FFT) may be used on the EKG, respiration, andblood pressure data to obtain the peek frequencies of EKG, respiration,and blood pressure, and then notch filers may be used to remove HCactivities on the ROIs with temporal frequencies centering around thesefrequencies. Independent component analysis (ICA) may be used toaccomplish the same goal.

Referring now to FIG. 10 an illustration of data-driven machine learningfor optimized hemoglobin image composition is shown. Using the filteredsignals from the ROIs of two or more than two emotional states 901 and902, machine learning 903 is employed to systematically identifybitplanes 904 that will significantly increase the signaldifferentiation between the different emotional state and bitplanes thatwill contribute nothing or decrease the signal differentiation betweendifferent emotional states. After discarding the latter, the remainingbitplane images 905 that optimally differentiate the emotional states ofinterest are obtained. To further improve SNR, the result can be fedback to the machine learning 903 process repeatedly until the SNRreaches an optimal asymptote.

The machine learning process involves manipulating the bitplane vectors(e.g., 8×8×8, 16×16×16) using image subtraction and addition to maximizethe signal differences in all ROIs between different emotional statesover the time period for a portion (e.g., 70%, 80%, 90%) of the subjectdata and validate on the remaining subject data. The addition orsubtraction is performed in a pixel-wise manner. An existing machinelearning algorithm, the Long Short Term Memory (LSTM) neural network, ora suitable alternative thereto is used to efficiently and obtaininformation about the improvement of differentiation between emotionalstates in terms of accuracy, which bitplane(s) contributes the bestinformation, and which does not in terms of feature selection. The LongShort Term Memory (LSTM) neural network or another suitable machinetraining approach (such as deep learning) allows us to perform groupfeature selections and classifications. The LSTM machine learningalgorithm is discussed in more detail below. From this process, the setof bitplanes to be isolated from image sequences to reflect temporalchanges in HC is obtained. An image filter is configured to isolate theidentified bitplanes in subsequent steps described below.

The image classification machine 105 is configured with trainedconfiguration data 102 from a training computer system previouslytrained with a training set of images captured using the above approach.In this manner, the image classification machine 105 benefits from thetraining performed by the training computer system. The imageclassification machine 104 classifies the captured image ascorresponding to an emotional state. In the second step, using a newtraining set of subject emotional data derived from the optimizedbiplane images provided above, machine learning is employed again tobuild computational models for emotional states of interests (e.g.,positive, negative, and neural).

Referring now to FIG. 11, an illustration of data-driven machinelearning for multidimensional invisible emotion model building is shown.To create such models, a second set of training subjects (preferably, anew multi-ethnic group of training subjects with different skin types)is recruited, and image sequences 1001 are obtained when they areexposed to stimuli eliciting known emotional response (e.g., positive,negative, neutral). An exemplary set of stimuli is the InternationalAffective Picture System, which has been commonly used to induceemotions and other well established emotion-evoking paradigms. The imagefilter is applied to the image sequences 1001 to generate high HC SNRimage sequences. The stimuli could further comprise non-visual aspects,such as auditory, taste, smell, touch or other sensory stimuli, orcombinations thereof.

Using this new training set of subject emotional data 1003 derived fromthe bitplane filtered images 1002, machine learning is used again tobuild computational models for emotional states of interests (e.g.,positive, negative, and neural) 1003. Note that the emotional state ofinterest used to identify remaining bitplane filtered images thatoptimally differentiate the emotional states of interest and the stateused to build computational models for emotional states of interestsmust be the same. For different emotional states of interests, theformer must be repeated before the latter commences.

The machine learning process again involves a portion of the subjectdata (e.g., 70%, 80%, 90% of the subject data) and uses the remainingsubject data to validate the model.

This second machine learning process thus produces separatemultidimensional (spatial and temporal) computational models of trainedemotions 1004.

To build different emotional models, facial HC change data on each pixelof each subject's face image is extracted (from Step 1) as a function oftime when the subject is viewing a particular emotion-evoking stimulus.To increase SNR, the subject's face is divided into a plurality of ROIsaccording to their differential underlying ANS regulatory mechanismsmentioned above, and the data in each ROI is averaged.

Referring now to FIG. 4, a plot illustrating differences in hemoglobindistribution for the forehead of a subject is shown. Though neitherhuman nor computer-based facial expression detection system may detectany facial expression differences, transdermal images show a markeddifference in hemoglobin distribution between positive 401, negative 402and neutral 403 conditions. Differences in hemoglobin distribution forthe nose and cheek of a subject may be seen in FIG. 6 and FIG. 7respectively.

The Long Short Term Memory (LSTM) neural network, or a suitablealternative such as non-linear Support Vector Machine, and deep learningmay again be used to assess the existence of common spatial-temporalpatterns of hemoglobin changes across subjects. The Long Short TermMemory (LSTM) neural network or an alternative is trained on thetransdermal data from a portion of the subjects (e.g., 70%, 80%, 90%) toobtain a multi-dimensional computational model for each of the threeinvisible emotional categories. The models are then tested on the datafrom the remaining training subjects.

These models form the basis for the trained configuration data 102.

Following these steps, it is now possible to obtain a video sequencefrom the camera 48 of a person in the vehicle 24 and apply the HCextracted from the selected biplanes to the computational models foremotional states of interest. The output will be a notificationcorresponding to (1) an estimated statistical probability that thesubject's emotional state belongs to one of the trained emotions, and(2) a normalized intensity measure of such emotional state. For longrunning video streams when emotional states change and intensityfluctuates, changes of the probability estimation and intensity scoresover time relying on HC data based on a moving time window (e.g., 10seconds) may be reported. It will be appreciated that the confidencelevel of categorization may be less than 100%.

Two example implementations for (1) obtaining information about theimprovement of differentiation between emotional states in terms ofaccuracy, (2) identifying which bitplane contributes the bestinformation and which does not in terms of feature selection, and (3)assessing the existence of common spatial-temporal patterns ofhemoglobin changes across subjects will now be described in more detail.One such implementation is a recurrent neural network.

One recurrent neural network is known as the Long Short Term Memory(LSTM) neural network, which is a category of neural network modelspecified for sequential data analysis and prediction. The LSTM neuralnetwork comprises at least three layers of cells. The first layer is aninput layer, which accepts the input data. The second (and perhapsadditional) layer is a hidden layer, which is composed of memory cells(see FIG. 13). The final layer is output layer, which generates theoutput value based on the hidden layer using Logistic Regression.

Each memory cell, as illustrated, comprises four main elements: an inputgate, a neuron with a self-recurrent connection (a connection toitself), a forget gate and an output gate. The self-recurrent connectionhas a weight of 1.0 and ensures that, barring any outside interference,the state of a memory cell can remain constant from one time step toanother. The gates serve to modulate the interactions between the memorycell itself and its environment. The input gate permits or prevents anincoming signal to alter the state of the memory cell. On the otherhand, the output gate can permit or prevent the state of the memory cellto have an effect on other neurons. Finally, the forget gate canmodulate the memory cell's self-recurrent connection, permitting thecell to remember or forget its previous state, as needed.

The equations below describe how a layer of memory cells is updated atevery time step ^(t). In these equations:

x_(t) is the input array to the memory cell layer at time . In ourapplication, this is the blood flow signal at all ROIs

r_(x) _(i) =[x_(1t) x_(2t) K x_(mt)]

-   -   W_(i), W_(f), W_(c), W_(o), U_(i), U_(r), U_(c), U_(o) and V_(o)        are weight matrices; and        -   b_(i), b_(f), b_(c) and b_(o) are bias vectors

First, we compute the values for i_(t), the input gate, and c^(%) _(t)the candidate value for the states of the memory cells at time t:

i _(t)=σ(W _(i)x_(t) +U _(i) h _(t-1) +b _(i))

C ^(%) _(t)=tan h(W _(c) x _(t) +U _(c) h _(t-1) +b _(c))

Second, we compute the value for f_(t), the activation of the memorycells' forget gates at time t:

f _(t)=σ(W _(f)x_(t) +U _(f) h _(t-1) +b _(f))

Given the value of the input gate activation i_(t), the forget gateactivation f_(t) and the candidate state value C^(%) _(t), we cancompute C_(t) the memory cells' new state at time t:

C _(t) =i _(t) *C ^(%) _(i) +f _(t) *C _(t-1)

With the new state of the memory cells, we can compute the value oftheir output gates and, subsequently, their outputs:

o _(t)=σ(W ₀ x _(t) +U ₀ h _(t-1) +V ₀ C _(t) +b ₀)

h _(t) =o _(t)*tan h(C _(t))

Based on the model of memory cells, for the blood flow distribution ateach time step, we can calculate the output from memory cells. Thus,from an input sequence x₀, x₁, x₂, L, x_(n), the memory cells in theLSTM layer will produce a representation sequence h₀, h₁, h₂, L, _(n).

The goal is to classify the sequence into different conditions. TheLogistic Regression output layer generates the probability of eachcondition based on the representation sequence from the LSTM hiddenlayer. The vector of the probabilities at time step t can be calculatedby:

p _(t)=softmax(W _(output) h _(t) +b _(output))

where w_(output) is the weight matrix from the hidden layer to theoutput layer, and b_(output) is the bias vector of the output layer. Thecondition with the maximum accumulated probability will be the predictedcondition of this sequence.

The computer system 36 registers the image streams captured from thecamera 48 and makes a determination of the invisible emotion detectedusing the process described above. The detected invisible emotion isthen registered with a time, date, license plate, and the video streamcaptured by the camera 48. The computer system 34 can be configured todiscard the image sequences upon detecting the invisible emotion. Thecomputer system 34 then notifies the border security guard 44 of thedetected invisible emotion and its intensity.

Referring now to FIG. 9, an exemplary screen 800 presented on thedisplay 40 by the computer system 36. The screen 800 presents a photo ofthe driver 28 retrieved from a database using the driver's passportidentification number, as well as various data related to the driver 28.In addition, the screen 800 presents a notification area 804 thatcomprises a color. The color shown corresponds to the detected invisibleemotion and its intensity. For a strong, positive invisible emotion, agreen field is presented. When a neutral emotion is detected, a whitefield is presented. If a negative invisible emotion is detected, a redfield is presented. The intensity of the color corresponds to theintensity of the detected invisible emotion. In the case of a negativedetected invisible emotion, the red field presented in the notificationarea 804 may flash to draw the attention of the border security guard44. As the display 40 is only visible to the border security guard 44,the driver 28 will be unaware of the detected invisible emotion.

The border security guard 44 can then use the presented notification todetect when the driver 28 may be ill at ease, potentially related todiscomfort with a question or with a deceptive answer provided by thedriver 28 in response to a question.

In other embodiments, the computer system 36 can be configured togenerate and present a graph of the detected invisible emotions so thatthe border security guard 44 can review past detected invisible emotionsin case the guard's attention was diverted.

In another embodiment, the computer system 36 can be configured topresent text on the display 40 to further notify a person of thedetected invisible emotion. In another embodiment, a separate devicesuch as an LED that is only visible to guard 44 can be employed in aposition in the field of view of the border security guard. In a furtherembodiment, the computer system 36 can notify a person of the detectedinvisible emotion via an audible noise transmitted to an earpiece wornby the person. In yet another embodiment, haptic feedback can beprovided to a person through a wearable device with a haptic engine andthat is in communication with the computer system 36. In still yetanother embodiment, a notification can be made on a surface of a pair ofglasses worn by a person that is only visible to the wearer.

The computer system 36 is configured in another embodiment to calculatea probability of deception by the person being questioned based on theemotions detected and their intensity, and present this informationgraphically and/or textually to the border security guard.

Other methods of representing the detected invisible emotion will beapparent to a person skilled in the art.

In another embodiment shown in FIG. 14, a kiosk 1100 enables an at leastpartially automatic screening process at a security checkpoint. Thekiosk 1100 may be, for example, provided in airport border controlstations. The kiosk 1100 has a touchscreen 1104 for presenting a seriesof questions in an interview via a display. Feedback can be received viathe touchscreen 1104, a keyboard 1108, a microphone (not shown), etc.The kiosk 1100 may also be equipped with a hidden camera 1110 that isconfigured to capture image sequences of the face of a person using thekiosk 1100. Alternatively, or in addition, a camera may be provided fora physical area corresponding to several kiosks and capable ofmonitoring a plurality of persons. A passport scanner 1112 scanspassports inserted therein.

The captured image sequences can be analyzed by a processor within thekiosk 1100 (or sent to another computing device for analysis) to detectinvisible human emotions, and thus deception, during the questionnaire.The kiosk 1100 may then prepare a summary of the interview responses,together with the invisible human emotions, and probability ofdeception, detected during the interview such that they are correlatedto the questions posed. In one configuration, the kiosk 1100 may notifysecurity personnel of a condition, such as a detected invisible humanemotion and corresponding probability of deception at a particular pointin the interview. In another configuration, the kiosk can print aninterview summary receipt via a printout slot having an identificationnumber or barcode corresponding with the results that are communicatedto and stored in a central database. The interviewed person can thentake the interview summary receipt to security staff for review. In thisconfiguration, the security staff can scan in the barcode or type in theidentification number from the interview summary receipt and review theresults retrieved from the central database.

In other embodiments, the camera capturing image sequences of a person'sface can be separate from the computing device that performs thedeception detection. The image sequences can be communicated to thecomputing device performing the detection via a wired or wirelesscomputer communications network, or via removable storage. For example,a smartphone can capture image sequences and audio and transmit themover a Wi-Fi network to a computer system that is configured to performthe invisible human emotion detection.

While the above-described embodiments are described in relation tocheckpoint security, it will be appreciated that the above method andsystem can be adapted for use with other types of security. For example,similar systems and methods can be used in airport security, buildingingress security, border/customs checkpoints, police investigations,military investigations, consular interviews, spying operations, courtdepositions, etc. Further, the system can be configured to monitor aperson of interest during other activities via one or more visible orinvisible cameras to detect invisible human emotions and thus deception.In various applications, the system can be used in connection with aquestion/answer methodology, to detect deception associated withspecific information, or with a candid capture methodology, to detectdeceptive intent generally.

Although the invention has been described with reference to certainspecific embodiments, various modifications thereof will be apparent tothose skilled in the art without departing from the spirit and scope ofthe invention as outlined in the claims appended hereto. The entiredisclosures of all references recited above are incorporated herein byreference.

We claim:
 1. A system for detecting deception for the security screeningof a person of interest by an attendant, the system comprising: a cameraconfigured to capture an image sequence of the person of interest; aprocessing unit trained to determine a set of bitplanes of a pluralityof images in the captured image sequence that represent the hemoglobinconcentration (HC) changes of the person, to detect the person'sinvisible emotional states based on HC changes, and to output thedetected invisible emotional states, the processing unit being trainedusing a training set comprising HC changes of subjects with knownemotional states; and, a notification device for providing anotification of at least one of the person's detected invisibleemotional states to the attendant based on the output of the processingunit.
 2. The system of claim 1, wherein the processing unit isconfigured to calculate a probability of deception of the person beingscreened based on the emotions detected and their determined intensity,and the notification device is configured to present the calculatedprobability of deception to the attendant.
 3. The system of claim 2,wherein the notification is configured to draw the attention of theattendant if the probability of deception exceeds a predetermined value.4. The system of claim 2, wherein the notification comprises a colorcoded visual indicator indicative of the detected invisible emotion andits intensity.
 5. The system of claim 4, wherein a positive invisibleemotion is depicted as green and a deceptive human emotion is depictedas red.
 6. The system of claim 3, wherein the notification comprises aflash to draw the attention of the attendant.
 7. The system of claim 3,wherein the notification comprises an audible noise.
 8. The system ofclaim 2, wherein the notification comprises a summary report of theprobability of deception during the screening of the person.
 9. Thesystem of claim 8, wherein the camera is further configured to capturean audio sequence in addition to the image sequence, and the processingunit is further configured to process the audio sequence to determine aset of questions posed to the person during the screening, and theprobability of deception in the summary report is visually correlated tothe questions.
 10. A method for detecting deception for the securityscreening of a person of interest by an attendant, the methodcomprising: capturing, by a camera, an image sequence of the person ofinterest; determining, by a processing unit, a set of bitplanes of aplurality of images in the captured image sequence that represent thehemoglobin concentration (HC) changes of the person, detecting theperson's invisible emotional states based on HC changes, and outputtingthe detected invisible emotional states, the processing unit beingtrained using a training set comprising HC changes of subjects withknown emotional states; and, providing a notification, by a notificationdevice, of at least one of the person's detected invisible emotionalstates to the attendant based on the output of the processing unit. 11.The method of claim 10, further comprising, calculating, by theprocessing unit, a probability of deception of the person being screenedbased on the emotions detected and their determined intensity, and,presenting, by the notification device, the probability of deception tothe attendant.
 12. The method of claim 11, wherein the notification isconfigured to draw the attention of the attendant if the probability ofdeception exceeds a predetermined value.
 13. The method of claim 11,wherein the notification comprises a color coded visual indicatorindicative of the detected invisible emotion and its intensity.
 14. Themethod of claim 13, wherein a positive invisible emotion is depicted asgreen and a deceptive human emotion is depicted as red.
 15. The methodof claim 12, wherein the notification comprises a flash to draw theattention of the attendant.
 16. The method of claim 12, wherein thenotification comprises an audible noise.
 17. The method of claim 11,wherein the notification comprises a summary report of the probabilityof deception during the screening of the person.
 18. The method of claim17, further comprising, capturing, by the camera, an audio sequence inaddition to the image sequence, and processing, by the processing unit,the audio sequence to determine a set of questions posed to the personduring the screening, and visually correlating, by the processing unit,the probability of deception to the questions in the summary report.